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Abstract 

Beyond Blackboards is an inquiry-centered, after-school program designed to enhance middle school students’ engagement with 
engineering through design-based experiences focused on the 21st Century Engineering Challenges. Set within a predominantly low- 
income, majority-minority community, our study aims to investigate the impact of Beyond Blackboards on students" interest in and 
understanding of engineering, as well as their ability to align their educational and career plans. We compare participants’ and 
nonparticipants’ questionnaire responses before the implementation and at the end of the program’s first academic year. Statistically 
significant findings indicate a school-wide increase in students’ interest in engineering careers, supporting a shift in school culture. 
However, only program participants showed increased enjoyment of design-based strategies, understanding of what engineers do, and 
awareness of the steps for preparing for an engineering career. These quantitative findings are supported by qualitative evidence from 
participant focus groups highlighting the importance of mentors in shaping students’ awareness of opportunities within engineering. 

Keywords: underrepresented minorities, robotics, after-school activities, Grand Challenges, design-based research, mixed-methods 


Introduction 

As many studies have demonstrated, our nation needs to take significant steps to develop our STEM labor force, 
particularly in engineering (Engel & Giddens, 2011; National Academies, 2010a, 2010b; National Science Board, 2010). 
Yet, a lack of engineering-specific curricula throughout the K-12 pipeline results in an ill-defined path toward engineering 
for students (Chandler, Fontenot, & Tate, 2011). Further, the U.S. student body is diverse and includes increasing numbers 
of low-income, minority, and immigrant students. Racial, ethnic, gender, and social-class differences in interest and 
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preparation appear as early as elementary and middle 
school, and early disadvantages produce persistent educa¬ 
tional and labor force disparities that limit the diversity of 
students entering STEM fields (Gonzales et al, 2008; 
Riegle-Crumb, Moore, & Ramos-Wada, 2011; Riegle- 
Crumb & King, 2010). These differences in engagement 
are especially problematic because middle-schoolers’ 
ability to align their educational plans and career goals 
may have long-term implications, particularly in STEM 
fields (Miller & Kimmel, 2012). Consequently, producing a 
larger and more diverse next generation of engineers 
requires that we improve underserved students’ interest in, 
knowledge about, and ability to track toward careers in 
engineering at this critical time in their educational 
experiences. 

This study investigates the impact of a research-based 
after-school program called Beyond Blackboards, a project 
rooted in the design principles of contextualized, inquiry- 
oriented instruction; socially relevant design problems; and 
collaborative problem solving. Beyond Blackboards uses 
mentored, designed-based inquiry to stimulate middle school 
students’ engagement with engineering and enable them 
to better align their educational plans with their occupational 
goals. We report on findings from the program’s first 
year within the Cave Creek School District (CCSD) in a 
predominantly low-income, Hispanic community in Texas. 1 
Although Cave Creek is just outside a metropolitan area with 
a booming, innovation-driven economy, 2 it is miles away 
in terms of its demographics, education, and occupational 
possibilities for students. As with many low-income and 
majority-minority communities in the USA, Cave Creek’s 
students and families have generally not had access to these 
opportunities, in part because they lack the education and 
skills required for technology-based careers. Thus, the 
salience of our analysis for increasing engagement and 
diversity in engineering is heightened by the geographic and 
economic context of the community in which our study 
took place. 

Our study targets understanding students’ attitudes and 
interests. We administered a questionnaire to students that 
addressed programmatic elements and also included items 
from the 2007 Trends in International Mathematics and 
Science Study (TIMSS), which allows us to empirically 
situate the relevance of our study by benchmarking the 
students in our study with their peers across the nation 
(International Association for the Evaluation of Educational 
Achievement [1EA], 2009). We then use an explanatory 
sequential mixed methods design (Creswell, 2014) to first 
investigate how the program impacts students’ interest in 
and knowledge about engineering as well as the alignment 
of their educational and career plans via quantitative 

1 The names of the school district and community have been masked. 

2 The adjacent city was declared “recession-proof" by Forbes Magazine 
shortly before our study began (Zumbrun, 2008). 


analysis before bringing qualitative evidence to bear in 
explaining our findings. Because such programs also have the 
potential to change the school culture by drawing attention 
and social prestige to students’ engineering accomplishments 
(Coleman, 1995), we consider the program’s impact on both 
program participants and their schoolmates. Taken collec¬ 
tively, our analysis contributes to understandings of (1) the 
engineering-related perspectives of middle school students 
from some of the most underrepresented populations in 
engineering, (2) how programs like Beyond Blackboards may 
shape participants’ knowledge about, interest in, and plans 
concerning engineering, and (3) the extent to which such a 
program may produce shifts in these areas school-wide, 
among participants and nonparticipants alike. 

Background 

Encouraging and Supporting Engineering Careers among 
Middle-Schoolers 

As students transition into middle school, many ex¬ 
perience a decline in positive attitudes about mathematics 
and science (Gonzales et al., 2008), and even before high 
school, girls and members of racial minorities are less 
likely than boys and white students to aspire to careers in 
mathematics and science (Riegle-Crumb et al., 2011). 
These attitudes can have long-term effects on students’ later 
career ambitions. Even at the elementary and middle school 
levels, schools can disrupt this downward trend by 
encouraging potential engineering students’ interests and 
talents and by giving them the preparation and resources 
they need to pursue education in engineering-related fields. 

However, many students do not know what engineers 
actually do or the role that engineers play in society. This 
lack of information may deter students from pursuing 
engineering degrees. Consistent with this argument, Jordan 
and Snyder (2013) found that even among participants in a 
middle school engineering club, “many students conceived 
of engineers as builders, repairpersons, or scientists. Few 
students identified more than two or three examples of what 
engineers design” (p. 1948). Conveying to students the 
important and dynamic role of engineers in society may 
increase students’ interest in engineering, particularly 
among women and underrepresented racial or ethnic 
minorities (National Academy of Engineering, 2008). 

Students may also lack an understanding of the types 
of skills and responsibilities required of engineers on a 
daily basis. In contrast to traditional mathematics and 
science courses, where students are typically encouraged 
to independently identify a single correct solution to 
problems, an engineering approach requires logical and 
divergent thinking, problem solving, communication of 
ideas, and teamwork. Along these lines, a National Science 
Foundation-published report recommends that engineers 
should possess “the ability to work in teams; a systems 
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approach to problem solving; greater knowledge and 
consideration of social, environmental, and other implica¬ 
tions of problem solving efforts; along with management 
skills and a capacity for life-long learning” (Coward, Ailes, 
& Bardon, 2000, p. 1). However, a study of students’ 
drawings of engineers and scientists found that middle 
school students’ views were antithetical to these recom¬ 
mendations; instead, they portrayed engineers as “doers” 
or “worker bees” rather than as problem-solving profes¬ 
sionals engaged in critical thinking (Fralick, Kearn, 
Thompson, & Lyons, 2009). Therefore, educational oppor¬ 
tunities that accurately highlight the skills needed for 
engineering may lead students to become more interested in 
engineering (National Academy of Engineering & National 
Research Council, 2009). 

As students develop their educational and occupational 
goals, exposure to different types of careers can help them 
identify the pragmatic steps to take to achieve the 
careers they want (Buck et al., 2008; Csikszentmihalyi & 
Schneider, 2000; Hamilton & Hamilton, 1997). However, 
even when students aspire to a particular career and gain 
the necessary academic preparation in middle and high 
schools, many still fall short of their goals. One explanation 
for this unsuccessful transition is that students who are 
unclear or unfocused about their occupational futures tend 
to over- or underestimate the amount of education they will 
need for the type of career they wish to pursue (Schneider 
& Stevenson, 1999). If students are to succeed at their 
goals, it is important that they have “aligned ambitions”— 
educational expectations that align with occupational 
aspirations (Schneider & Stevenson, 1999). Students’ 
ability to realize their career goals can be stifled if they 
are unable to clearly articulate or understand the pathways 
from secondary through postsecondary school and into 
careers. A recent study of long-term labor market outcomes 
found that young people who had underestimated the 
education required for their future occupations achieved, on 
average, lower levels of education and lower wages than 
those with high and aligned ambitions (Sabates, Harris, & 
Staff, 2011). These findings demonstrate that it is important 
for students’ educational ambitions to align with their 
occupational goals. 

While it is important for all students to be able to align 
their educational trajectory with their occupational goals, 
there is some evidence that this alignment may be especially 
important for STEM careers. Research suggests that 
potential STEM students often have difficulty identifying 
realistic strategies that will help them achieve their career 
goals. For example, students often do not know how to 
select coursework and extracurricular activities appropriate 
for STEM careers (Csikszentmihalyi & Schneider, 2000; 
Schneider & Stevenson, 1999). Additionally, middle school 
represents a critical point in which students are sorted into 
different math course levels based on their interests and 
performance—a mechanism that has long-term implications 


for their preparedness for college, STEM fields, and 
engineering in particular (Dauber, Alexander, & Entwisle, 
1996; Miller & Kimmel, 2012). Engineering interventions 
and outreach need to help students develop realistic 
strategies to bridge their interests with the opportunities 
available to them during the complex transitional years of 
middle and high school. 

The Call for Strategic Interventions 

Schools must play a critical role in addressing these 
issues. Not only do schools have the capacity to increase 
students’ knowledge about what engineering is, how 
engineers work, and how to align their educational and 
career ambitions, but schools can also influence students by 
creating a culture that promotes engineering. The National 
Academy of Engineering (2008) recommends that schools 
strategize to deliver messages that will affect how students 
perceive and evaluate STEM fields. For example, robotics 
competitions can positively influence participants’ attitudes 
toward science (Welch & Huffman, 2011). Schools can 
also promote students’ interest in and knowledge about 
engineering careers by shifting the curricular emphasis 
from mathematics and science achievement to proficiency 
in the types of skills required of engineers (National 
Academy of Engineering & National Research Council, 
2009; Oware, Capobianco, & Diefes-Dux, 2007). These 
types of changes can improve potential engineering 
students’ attitudes toward academic pursuits (Anderson, 
1982; Thapa, Cohen, Guffey, & Higgins-D’Alessandro, 
2013) and can even encourage them to put forth additional 
effort in school (Schunk, Pintrich, & Meece, 2008). 

Several recent publications from the National Acad¬ 
emies, National Science Foundation, and National Science 
Board address the declining competitiveness of the United 
States in the face of an increasingly technology-driven 
global economy. Each report describes ways that school 
climate can contribute to the production of engineers to 
meet the needs of this changing economy. While Rising 
Above the Gathering Storm (National Academies, 2007) 
emphasized the need for more inquiry-based learning in 
public schools, its 2010 update argued that developing a 
STEM workforce requires a “learning ecosystem” that 
includes school, family, and community members who 
cultivate and celebrate intellectual achievement (National 
Academies, 2010b). Combating low expectations, “math 
phobia,” and peer hostility requires additional support for 
students and improved preparation for administrators and 
teachers (National Academies, 2010b). 

Furthermore, K-12 schools can give students a more 
accurate and motivating impression of engineering by 
changing the ways that they portray engineering careers. 
Specifically, students are responsive to messages about the 
social importance of engineers as links between society and 
technology (National Academy of Engineering, 2008). 
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Particularly in the case of underrepresented minorities, 
interventions require adapting formal and informal science 
content to the culturally specific forces within a school 
(National Science Board, 2010). The Beyond Blackboards 
program was designed to address these recommendations 
by using the National Academy of Engineering 21st 
Century Grand Challenges to frame the curriculum. 3 
Alongside their teachers and mentors, students explore 
and design engineering solutions for some of our world’s 
most pressing social issues. Their Beyond Blackboards 
experiences are intended to help students get a sense of 
what their own engineering careers could be like. 

School interventions can provide students with pathways 
into STEM fields. 4 Several studies have demonstrated that 
engineering outreach programs can improve students’ 
mathematics and science content knowledge (Hotaling, 
2007; McKay & McGrath, 2007; Mooney & Laubach, 
2002; Schaefer, Sullivan, & Yowell, 2003). Other studies 
have demonstrated a link between such programs and 
students’ learning and ability to explain, evaluate, or predict 
engineering concepts, processes, and outcomes (Barnett, 
2005; Satchwell & Loepp, 2002). This study examines how 
the Beyond Blackboards program contributes to students’ 
interest in and understanding of engineering as well as the 
extent to which students are able to align their educational 
and career ambitions. We also investigate the effect of the 
program on the school culture by considering these changes 
among nonparticipating peers. 

Design-based Research, Cave Creek, and the 
Beyond Blackboards Program 

Beyond Blackboards is an example of design-based 
research. According to Wang and Hannafin (2006), we 
define design-based research as “a systematic but flexible 
methodology intended to improve educational practices 
through iterative analysis, design, development, and 
implementation, based on collaboration among researchers 
and practitioners in real-world settings, and leading to 
contextually-sensitive design principles and theories” (p. 6; 
see also Brown, 1992; Design-Based Research Collective, 
2003). Educational researchers have argued that failing to 
consider the complexities of the learning context overstates 
the utility of experimental educational research; by 
contrast, design-based research takes an in vivo approach 
to producing plausible causal accounts within particular 
settings (Design-Based Research Collective, 2003; Johri & 
Olds, 2011). Numerous studies have demonstrated the 
effectiveness of this approach in generating theoretical 
understanding of the impact of interventions in K-12 


3 http://www.engineeringchallenges.org/ 

4 For a comprehensive overview, see Jeffers, Safferman, and Safferman 

(2004). 


engineering education (for example, Chiu & Linn, 2011; 
Marshall & Berland, 2012). The complexity of the Beyond 
Blackboards program, including the format of the program; 
the particulars of the local context; the multiple university, 
industry, school-district, and student actors; and the 
curriculum itself make it well suited for a design-based 
research approach. We now discuss each of these aspects in 
more detail. 5 

In fall 2010, Beyond Blackboards launched after-school 
robotics and engineering clubs at the CCSD’s three middle 
schools. Teachers were recruited and provided with 
professional development to teach design-based learning 
and coach the clubs to compete in a FIRST LEGO League 
(FLL) competition. Each club had supporting under¬ 
graduate college students as well as industry mentors. 
Less than four months after the program began, student 
teams competed in a robotics competition, performed well, 
and in some cases won awards. The teams’ successes drew 
media attention, which was heightened by the district’s 
decision to host a regional qualifying competition. This 
attention generated community-wide pride over the accom¬ 
plishments of the district’s students. 

The middle schools of CCSD offer a case study of some 
of the most underrepresented populations in engineering. 
Encompassing a cluster of seven small rural communities, 
the district has grown in recent years, primarily due to 
increasing numbers of low- and moderate-income Hispanic 
families. The school district reflects this shift: four out of 
five students are Hispanic, only 12 percent are black, and 5 
percent are white. One in three students is classified as 
limited English proficient (LEP), and 87 percent of students 
are economically disadvantaged. Before the Beyond Black¬ 
boards program began, the district did not offer any 
outreach programs or informal educational experiences that 
would allow middle school students to become involved in 
STEM-related learning outside of the standard curriculum 
mandated by the State of Texas. Cave Creek students are an 
ideal target population for efforts to increase diversity in 
engineering and improve U.S. competitiveness. 

CCSD collaborates with the engineering and education 
schools of a nearby university in a unique, ongoing 
partnership designed to implement the after-school robotics 
clubs and investigate their impact via design-based research. 
Open to all students, the Beyond Blackboards program 
includes after-school “Innovation Clubs” that were first 
implemented in the 2010-2011 academic year during the fall 
and spring semesters. All components of the Beyond 
Blackboards program present students with engineering- 
design challenges using LEGO® MINDSTORMS NXT 
robotics kits and the NXT-G programming environment. At 
each school, the Innovation Clubs met twice a week and 


5 For an overview of the approach to engineering education and more 
specifics regarding the curriculum of Beyond Blackboards, see the 
Appendix. 
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while the length of each meeting varied, the Innovation Club 
at each school had a total of 48 program hours during each 
semester. The research reported here focuses on the 
Innovation Clubs’ first academic year. 

Several community and university actors were involved 
in the implementation of the Beyond Blackboards program. 
The team from the university attended after-school activity 
nights organized by school personnel to introduce robotics 
and Innovation Club and to inspire student interest. Next, 
district and school staff coordinated club meeting schedules 
and student attendance while the team from the engineering 
school provided program materials and teacher training. 
During the 2010-2011 school year, the Innovation Clubs 
were led by two teachers at each of the three district middle 
schools. Prior to leading the clubs, these six teachers 
attended a professional development institute (PDI) 
facilitated by faculty members and graduate students at 
the university during the summer of 2010. At this PDI and 
at two follow-up PDIs during the school year, teachers 
received instruction and practice in design-based learning, 
the 21st Century Grand Challenges, automation and 
control, and how to construct and program robots using 
the LEGO products. 

Additionally, undergraduate students served as technical 
mentors and role models for the students. The mentors 
attended orientation meetings to understand their roles and 
mentoring strategies and to develop expertise in LEGO 
robotics. Engineers from local companies served as 
industry mentors and worked alongside the teachers and 
undergraduates to provide support and a professional 
perspective. The presence of industry mentors often 
entailed exposure to engineering careers for the students. 
Through coordination among the Innovation Club teachers, 
the university, and community organizations, speakers from 
local companies also visited the schools to raise awareness 
of STEM career paths among students in Cave Creek. 

In fall 2010, the Innovation Clubs coincided with the 
FLL season. FLL is a robotics competition affiliated with 
FIRST Robotics for students aged 9 to 14. Teams of up to 
ten students design LEGO robots to accomplish “missions” 
that are built around a particular societal need—for 
example, clean energy, food supply, or biomedical 
engineering. Teams receive points based on the number 
of missions they complete during two-and-a-half-minute 
trials. In addition, teams complete a research project related 
to the competition’s theme and engage in community 
outreach to propose their solution. Each team is judged not 
only by the success of their missions, but also by the 
quality of their research project, their robot design, and 
their demonstration of FLL core values, including team¬ 
work and gracious professionalism. 

During fall 2010, Beyond Blackboards supported two 
teams from each of the three CCSD middle schools. 
Innovation Club participants were introduced to LEGO 
MINDSTORMS NXT through a series of directed 


robot-building laboratories and scaffolded instruction in 
NXT-G programming. After this introduction, teachers 
discussed strategies for prioritizing the missions for 
competition and explained the technical requirements for 
each mission. Teachers guided students as they designed 
their competition vehicles, and students chose research 
projects and developed their presentations for the competi¬ 
tion. During the 2010 FLL season, all six teams 
participated in a qualifying competition, and three of the 
six teams were invited to participate in the regional 
championship. Also, one middle school in the district 
hosted a qualifying competition. As the season went on, the 
district began to embrace and celebrate the program and the 
students in the Innovation Clubs. 

During spring 2011, the Innovation Clubs transitioned 
into a program developed by the team at the university 
around two of the National Academy of Engineering 21st 
Century Grand Challenges. For example, in “(Re)New 
Orleans,” the program’s curriculum guided students in 
designing solutions to restore and improve urban infra¬ 
structure using robotics. As a second example, in “Exploring 
Alternative Energy,” students explored solar and wind 
power generation using LEGO MINDSTORMS NXT 
combined with other materials, including solar panels, wind 
turbines, and power meters. 

Beyond Blackboards is motivated by the principle that 
all students can benefit from exposure to an engineering 
curriculum. Although not all students will enter the field 
of engineering, students gain problem-solving experi¬ 
ences, tolerance for ambiguity, understanding of systems 
thinking, and technological fluency (Partnership for 21st 
Century Skills, 2011). Through the Beyond Blackboards 
program and in conjunction with events organized by the 
school site for the community, CCSD parents also have 
the opportunity to engage in engineering concepts and 
activities designed by the university team as a part of the 
program’s outreach component. Beyond Blackboards is an 
integrated program that gives middle school students, 
teachers, administrators, parents, and caregivers opportu¬ 
nities to participate in activities that will improve their 
understanding of STEM college and career pathways. 

Research Objectives 

Our study investigates the impact of the first year of the 
Beyond Blackboards program. Our research questions 
include: (1) In what ways does the program shape students’ 
interest in, knowledge about, and understanding of how to 
prepare for a career in engineering among participants and 
nonparticipants? (2) How do students’ descriptions of their 
own experiences account for these patterns? 

We proceed in three parts. First, we empirically situate 
the relevance of our study by benchmarking the students of 
CCSD against the nationally representative sample in the 
TIMSS according to their backgrounds, school attitudes, and 
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educational expectations. Next, we quantitatively compare 
Innovation Club participants to their nonparticipant school¬ 
mates to investigate changes in students’ attitudes toward 
school, interest in and knowledge about engineering, and 
educational expectations following participation in Beyond 
Blackboards. Last, we bring students’ own words to bear as we 
integrate qualitative data from focus groups into understanding 
how participation in Innovation Club may have shaped 
changes in students’ interest in, knowledge about, and ability 
to align their educational plans with engineering career goals. 

Based on our review of the literature in this area, we 
hypothesize that at an age when students’ attitudes toward 
school generally worsen, participants will have heightened 
interest in and understanding of careers in engineering. 
Further, we expect school-wide effects such that the 
Beyond Blackboards program will enhance the prestige of 
STEM careers among nonparticipants, but that it will not 
improve their understanding of engineering careers or how 
to track toward them. 

Data and Methods 

Data Collection 

The data for the quantitative component of this study were 
collected via surveys containing general academic items 
borrowed from the TIMSS (IEA, 2009) as well as original 
items concerning engineering interests, knowledge, and 
plans. The Beyond Blackboards research team collected 
these data during the 2010-2011 academic year, the first 
year of the program. CCSD includes three middle schools 
serving approximately 2,200 students that feed into the same 
high school; all three schools participated in the program and 
the data collection. All middle school students were targeted 
for a survey in September 2010, before the program began, 
and again in May 2011, after completion of the district’s 
state-mandated standardized testing schedule. The research 
was approved by the university’s Institutional Review Board 
and the school district. The pencil-and-paper questionnaires 


were short (approximately 15-20 minutes long), offered in 
English and Spanish, and completed by students on 
scannable answer sheets like the ones used by the district 
for benchmark testing. Student survey responses were then 
scanned into the district’s data system, which automatically 
links with the district’s administrative database. 

According to year-end administrative records, 2,200 
middle schoolers were enrolled in 2010-2011. Of these, 
1,943 and 1,835 students completed the fall and spring 
surveys, respectively. Coaches reported on students’ parti¬ 
cipation in Beyond Blackboards. Of the 80 participants 
listed, 74 were linked to their administrative records and 58 
participants were surveyed in both the fall and the spring. 
Most non-completers were absent from school, not enrolled 
at the time of one of the surveys, or otherwise unavailable to 
sit for the questionnaire on the survey day. Table 1 shows 
the demographic characteristics of CCSD middle-schoolers 
compared with Innovation Club participants; this informa¬ 
tion is based on district records. Hispanic students were two- 
thirds of the participants, 18 percent of participants were 
white, and 14 percent were black. Three-quarters of 
participants were economically disadvantaged, 30 percent 
were female, and 27 percent were designated as LEP. 
Although Hispanics, girls, and economically disadvantaged 
students were underrepresented in the program, it included a 
relatively large number of these underrepresented students. 

The questionnaire included items about students’ interest 
in and understanding of engineering, attitudes toward 
school, and educational expectations. Students’ interest in 
STEM careers and engineering in particular were measured 
using questions about students’ interest in careers as 
scientists or engineers. Other items measured students’ 
general interest in engineering (“I like learning how things 
work”), and understanding of how engineers work 
(“Engineers mainly work with other people to solve 
problems”). School attitudes were measured using stu¬ 
dents’ responses to questions about whether they liked 
school, thought that most students tried their best, and 
thought that most teachers wanted students to do their best. 


Table 1 

Demographic information for Cave Creek middle schools. 



Cave Creek middle schools 

Beyond Blackboards participants 

P 

Race/ethnicity (%) 

White 

5 

18 

*** 

Black 

12 

14 


Hispanic or Latino/a 

81 

66 


Other 

2 

2 


Student background characteristics (%) 

Female 

49 

30 

*** 

Economically disadvantaged 

87 

76 

** 

Limited English proficiency 

32 

27 


Number of students 

2220 

74 



***p <0.001, ** p <0.01, * p <0.05, + <0.10. 

Source: 2010-2011 Cave Creek School District administrative records and Beyond Blackboards participation records. 
Note: Race/ethnic differences are established using Pearson’s y 2 test. All other differences are tested using a t-test. 
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All attitude measures asked students to respond “agree a 
lot,” “agree a little,” “disagree a little,” or “disagree a 
lot.” In analysis, the positive and negative responses were 
recoded as “agree” or “disagree.” Last, students reported 
the highest level of education they expected to complete: 
finish high school, a two-year degree, a four-year degree, or 
an advanced degree; they could also indicate that they did 
not know. The questionnaire also included several items 
about students’ family background, including their parents’ 
level of education, whether they were bom in the United 
States or elsewhere, and how often they used a language 
other than English at home. 

Additionally, we use questionnaire items from the nationally 
representative, U.S. sample in the TIMSS. Although TIMSS 
includes measures of math and science attitudes, these pertain 
more closely to the math and science curriculum and class¬ 
room practices than to engineering (Gonzales et al., 2008). 
There were no references to engineering in TIMSS, nor were 
there questions about wanting a career in math or science. 
By contrast, the Beyond Blackboards surveys incorporated 
original measures of engineering attitudes and interest based 
on the program’s focus on innovation, understanding how 
things work, and teamwork. 

Focus Groups 

The research team also conducted focus groups toward 
the end of the school year. Student focus groups were 
conducted at each school and lasted about one hour. The 
research team worked with the Innovation Club teachers to 
coordinate the student focus groups. A total of 19 students 
who were club participants near the end of the school year 
were able to take part in the focus groups; about one half of 
the students had been involved in the program since the fall 
semester. A focus group that lasted about an hour with five 
of the six teachers was conducted at the university and 
followed a half-day professional development session. The 
focus groups were not recorded; extensive notes were taken 
by at least one researcher and a research assistant during 
each focus group and were reviewed to add details 
immediately after each group. The focus groups followed 
an interview protocol aimed at understanding the students’ 
and teachers’ perspectives about the program and its 
impact. Students were asked about their motivation for 
participation, their engagement with the club, their 
perceptions of how their peers see the club, and how 
participation shaped their views of school, engineering, and 
problem solving. Every participant in each focus group had 
the opportunity to share their views. In the student focus 
groups, special attention was given to quieter participants 
as a way to encourage their input. 

Analytical Plan 

To contextualize the CCSD, we begin our analysis by 
comparing CCSD students with a nationally representative 


sample from the TIMSS. The TIMSS was administered to 
eighth-graders in the spring; to gain the most accurate 
comparisons, we use only eighth-grade CCSD students’ 
responses from the spring survey. Using statistical tests, we 
compare Cave Creek students to their national counterparts 
according to their family background characteristics, race 
and ethnicity, school attitudes, and educational expectations. 

Next, the quasi-experimental design of this study allows 
us to contrast the program participants with peers at the 
same school according to their school attitudes, engineer¬ 
ing attitudes, and educational expectations. We compare 
participants’ and nonparticipants’ attitudes and expecta¬ 
tions in the fall and spring, using one-sample f-tests for the 
dichotomous attitudinal measures and a chi-square test for 
students’ educational expectations. Next, we estimate the 
mean change for the attitudes of participants compared with 
nonparticipants, and use a paired-sample f-test to estimate 
whether the average student change from fall to spring is 
significantly different from zero. While statistical tests for 
changes over time necessitate reducing our sample to only 
those students who were surveyed in both fall and spring, 
we report the fall and spring attitudes of all survey takers. 
This allows for the most accurate portrait of the CCSD 
student body at various points in time and allows future 
researchers to benchmark their data against our findings. 
We use a Friedman test to contrast the spring educational 
expectations with those from the fall for both participants 
and nonparticipants. 6 

Finally, we bring the qualitative evidence from our focus 
groups to bear to explain in students’ own words how 
participation in Innovation Club may have shaped their 
interest in and understanding of engineering careers and 
their educational plans. This approach is consistent with an 
explanatory sequential mixed methods design in which 
qualitative analysis elaborates and expands on a set of 
quantitative findings (Creswell, 2014). To do this, two 
researchers from the team independently identified themes 
by open coding the interview records with the purpose of 
identifying aspects of participation that may have influ¬ 
enced students over the course of the year (Merriam, 2009; 
see also Knaggs, Sondergeld, & Schardt, 2015). After an 
iterative process, the researchers settled on the three themes 
described in the results section. 

Results 

Benchmarking Cave Creek against the U.S. TIMSS 2007 

Before investigating the effect of Beyond Blackboards, 
we compare the eighth-grade CCSD students with the 


6 We replicated our analysis for the subsample of students who completed 
both the fall and spring surveys and found no differences in our results or 
interpretation. Therefore, to provide the clearest descriptive picture of our 
participants and nonparticipants in fall and spring, we include the full 
cross-sections in our analyses of students’ fall and spring reports. 
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Table 2 

Middle school student background and attitudes toward school. 



Eighth-graders 

Cave Creek 

TIMSS 

P 

Student background characteristics (%) 




Parent has a high school degree 

62.74 

88.25 

*** 

Parent has an associate’s or bachelor’s degree 

35.97 

53.82 

*** 

Student bom in the U.S. 

82.90 

90.36 

*** 

Student always speaks English at home 

20.86 

75.62 

*** 

Race/ethnicity (%) 



*** 

White 

6.28 

55.15 


Black 

12.72 

12.41 


Hispanic or Latino/a 

79.71 

23.38 


Other 

1.29 

9.06 


School attitudes (% agree) 




I like school 

74.22 

72.77 


I think most students try to do their best 

60.36 

52.98 

*** 

I think most teachers in my school want students to do their best 

90.64 

88.65 


Educational expectations (%) 



*** 

Finish high school 

11.18 

6.90 


Complete a two-year degree 

7.89 

8.39 


Complete a four-year degree 

31.25 

34.71 


Complete an advanced degree 

38.82 

41.30 


Don’t know 

10.86 

8.69 


Number of students 

621 

7377 



***> <0.001. ** p <0.01, * p <0.05. + <0.10. 

Source: Beyond Blackboards spring 2011 survey and TIMSS. 

Note. Student background characteristics and school attitude differences are tested using a /-Lest and race/ethnic and educational expectations differences are 
established using Pearson’s y 2 test. 


sample of U.S. eighth-graders in TIMSS 2007. These 
results are reported in Table 2. The left-hand column shows 
the spring survey responses of CCSD eighth-graders. 
Compared with the nationally representative TIMSS 
students, Cave Creek students had parents with less 
education and were more likely to have been born outside 
of the United States. Only 21 percent of the students 
reported speaking only English at home, compared with 76 
percent of the national sample. CCSD eighth-grade 
respondents are predominantly Hispanic (80 percent), 
compared with 23 percent in the national sample. 
Nonetheless, roughly three in four CCSD eighth-graders 
reported that they like school. While 91 percent of students 
reported that they felt that the teachers at their school 
wanted students to do their best, only 60 percent of all 
students agreed that their fellow students did their best in 
school. Although CCSD students differ significantly from 
their nationwide peers on background characteristics 
related to parents’ education, nativity, ethnicity, and 
language use at home, there are no differences between 
CCSD and TIMSS students with respect to liking school 
and thinking that their teachers want them to do their best. 
In fact. Cave Creek students hold more positive attitudes 
about their peers’ effort as they are more likely to report 
that students try their best. 

The bottom part of Table 2 reports educational expecta¬ 
tions for CCSD and TIMSS students. The largest share of 


CCSD middle school students (39 percent) reported intend¬ 
ing to earn an advanced degree such as a master’s, JD, or 
PhD; however, this figure is slightly lower than the national 
average (41 percent). Overall, a larger share of CCSD 
students than TIMSS students do not expect to complete a 
four-year degree or higher. The chi-square test revealed that 
the difference in educational expectations between CCSD 
students and the TIMSS sample is statistically significant, 
and that the main differences are that CCSD students are 
more likely either to not know their educational expectations 
or to only expect to finish high school. 

Comparisons between Participants’ and Nonparticipants’ 
Attitudes over Time 

Turning to the analysis of CCSD students and changes in 
their attitudes between fall and spring. Table 3 shows 
differences between participants and nonparticipants in fall 
and spring on the left and changes within these two groups 
over this period on the right. In fall, students who 
participated in the program were more likely than 
nonparticipants to like school. In spring, participants were 
less likely than nonparticipants to report that most students 
try their best. In general, school attitudes declined between 
the fall and spring. The mean changes over the year 
indicate that on average, students experienced a downward 
shift in thinking their peers tried to do their best at school. 
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Fewer participants reported that they liked school in the 
spring compared with the fall, and nonparticipants declined 
in their perceptions that teachers in their school wanted 
students to try their best over the course of the school year. 

Students were also asked about engineering-specific 
attitudes. The left-hand side of Table 3 indicates that 
participants, compared with nonparticipants, were more 
interested in careers in science and engineering before 
they even enrolled in the Beyond Blackboards program. In 
spring, nonparticipants showed greater interest in becom¬ 
ing a scientist or engineer, and participants indicated 
greater interest in engineering only. Furthermore, whereas 
nonparticipants adopted more negative attitudes about 
learning how things work and demonstrated diminished 
understanding of the teamwork involved in engineering 
careers, Beyond Blackboards participants maintained their 
attitudes and understanding from the fall. These trends 
are confirmed by the measures of mean change on the 
right-hand side of Table 3. This suggests that the program 
may play a role in protecting participants from the 
usual decline in interest and attitudes, and that the 
program may have enhanced participants’ interest in 
becoming engineers. 

Last, the bottom of Table 3 shows differences between 
nonparticipants’ and participants’ educational expectations 
in the fall and in the spring. Although there are no fall 
differences, in spring more participants expected to earn a 
four-year degree or higher and fewer participants were 
unsure of their expectations (center column). No statistical 
differences were observed in the educational expectations of 
nonparticipants between spring and fall. However, within 
our small sample of participants, changes in educational 
expectations approach significance (p = 0.11). Two notable 
trends appear among the participants. First, in the fall, 
participants reported especially high levels of not knowing 
their educational expectations. This category is markedly 
reduced by the spring, suggesting that participants honed 
expectations over the academic year. A similar pattern is 
apparent for participants who began the year only 
expecting to graduate from high school; these students 
appear to raise their expectations during the academic 
year. Strikingly, participants’ reports of “don’t know” (18 
percent) in fall far exceeds the TIMSS average for eighth- 
graders (shown in Table 2 as 9 percent), and drops below 
it in spring (5 percent). These patterns of change are not 
apparent among nonparticipants. The second notable trend 
distinguishing participants is that they showed a striking 
shift to reporting that they expect to complete a four-year 
degree, typically required for many engineering jobs, 
rather than an advanced degree. This is consistent with 
planning for a career in engineering which was reported 
by the majority of participants (71 percent). The shift in 
educational expectations for nonparticipants is simply 
toward an advanced degree, though a substantial por¬ 
tion of nonparticipants did not know their educational 


expectations or planned to only complete high school in 
the spring. 

Unpacking the Program’s Impact: Evidence from Focus 
Groups 

To better illustrate why experiences in Beyond Black¬ 
boards affected students’ interest in, knowledge about, and 
alignment of educational plans and career goals, we consider 
students’ and teachers’ own words about their experiences 
shared in focus groups. Three related themes that appeared 
consistently were: students’ broadened perspectives of what 
engineers do, participants’ increased motivation and con¬ 
fidence about their own engineering abilities, and the role of 
mentors in producing these changes. 

Consistent with the quantitative analysis, participating in 
Innovation Club increased students’ interest in engineering 
and their understanding of the link between engineering 
and innovation. One student offered, "When you see 
something you made or see something you worked on, you 
feel proud.” Participants responded favorably to the 
teamwork, creativity, and challenges involved in competing 
against other teams. Another student described the excite¬ 
ment of competitions saying, “You got [to] see other teams, 
see their builds, [and] get new ideas for new builds. We all 
came together as a team. We were nervous. Adrenaline!” 
Participating in Innovation Club allowed participants to 
engage with supportive mentors in a different capacity than 
they normally experience in their classrooms. When 
describing their coaches, students reported how their 
mentors “were fun because they were helpful and show us 
what we did wrong.. .but you have to figure it out yourself.” 
Students’ new understandings of engineers as innovators 
clearly resonated with them; one student reflected that, “if 
people would actually take the time to learn how things 
work, we could probably fix a lot of stuff.” 

In addition to increased interest in engineering, participat¬ 
ing in Beyond Blackboards broadened students’ views of 
engineering opportunities and of their own potential as 
engineers. For example, one student reported that at first she 
wanted to be a teacher, but that after learning more about 
robotics from her coach, she now wants to be a chemical or 
civil engineer since “I’ve always liked building stuff, like 
with LEGOs.” Another student described the appeal of the 
global opportunities associated with engineering: “I want to 
be an engineer because the speaker [an industry mentor] 
told us that engineers travel and I really want to go to 
Japan.” Students also learned about their own interests 
and skills. A student reported that while participating, he 
found out that he was good with tools and now wants to be 
an engineer because “[he likes] building with wood, 
LEGOs, and other materials.” However, some students 
were broadminded when describing how to apply their 
new skills. One student wanted to be “an FBI agent. I’m 
very curious about learning how things work, so I guess 
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that would help with being in the FBI since it’s a mystery 
and you have to figure it out.” In all, evidence from the 
focus groups suggests that students may have increased 
their interest in becoming engineers through learning more 
about engineering as well as their own strengths via 
participation and their mentors. 

Participants in the program were not only developing 
long-term aspirations, but were adopting concrete strategies 
that can support their goals while they are in school— 
factors that may explain a shift in educational expectations 
among participants observed in the quantitative analysis. 
Multiple students reported plans to take pre-advanced 
placement (pre-AP) science and math. One girl in eighth 
grade explained, “I got better at math. The mentors (from 
the university) talk about rotation and distance of things. 
I’m in a regular math class, but I have a 100... Next year, I 
am taking Algebra.” In addition, students also gained an 
enhanced understanding of engineering skills such as 
problem solving and critical thinking. One student reported 
improving at problem solving which “helped [her] focus 
more in school... [like] with answering questions.” A 
teacher coach noted that, “I had a child that was trying to 
do a build, and he was getting frustrated... [When] he 
finally got it, his robot was working and he was absolutely 
thrilled.” She added that. 

Through robotics, students learned problem solving and 
analytical thinking skills. The higher-level thinking, it’s 
everywhere; the government wants us to go with higher- 
level thinking and STEM.... College readiness? Yeah! 
These kids will be problem ready, [ready] for industry, 
teamwork. 

Beyond Blackboards may shape middle school students’ 
goals not only by exposing them to engineering, but also by 
enhancing their confidence in their ability to solve 
problems and to be academically competitive in high 
school and beyond. 

Discussion and Conclusion 

Beyond Blackboards represents a possible response to 
the call to broaden access to STEM fields and increase 
awareness of the engineering profession, especially for 
traditionally underrepresented groups (National Science 
Board, 2010). The CCSD serves underrepresented students 
who could take advantage of the rich opportunities in the 
larger metro area if they gain the education and training to 
do so. Our comparison with the national TIMSS 2007 
sample demonstrates that although the CCSD students’ 
attitudes toward school are similar to or, in some cases, 
more positive than those of their national peers, they have 
lower educational expectations. Beyond Blackboards is 
designed to encourage students to engage with engineering 
and pursue relevant educational opportunities. The 


approach is systemic in that it is designed to engage the 
entire school community and stimulate interaction between 
parents, industry mentors, undergraduates, district admin¬ 
istrators, and other community members. The goal of these 
activities is to pave the way for students to embrace 
engineering, consider engineering careers, and align their 
educational and occupational ambitions. 

We set out to investigate the impact of the Beyond 
Blackboards program on three outcomes critical to the 
engagement of underserved students with engineering. 
Specifically, we examined whether Beyond Blackboards 
increased participants’ interest in, knowledge about, and 
understanding of how to prepare for a career in engineer¬ 
ing. Our findings show that Beyond Blackboards partici¬ 
pants, who were initially more interested than their 
schoolmates in engineering careers, gained even greater 
interest over the academic year. Further, we document a 
trend of increased interest in four-year degrees over 
undefined or highly ambitious postgraduate degrees among 
participants, which suggests that these students were 
aligning their ambitions around the skills required of them 
to pursue careers in engineering. Although these early 
results are suggestive of program impact rather than 
conclusive, our qualitative data from focus groups support 
that participants have changed the way they think about 
engineering. 

Further, we examined whether the program’s impact 
might extend to nonparticipants as well. We found possible 
evidence of this systemic impact on the broader school 
culture. During adolescence, when students’ attitudes toward 
school typically decline (Riegle-Crumb et al., 2011), CCSD 
students experienced positive changes in attitudes toward 
STEM careers in general and engineering in particular. 
Strikingly, both participants’ and nonparticipants’ enthu¬ 
siasm increased, suggesting that middle-schoolers’ enthu¬ 
siasm for science and engineering appears to have increased 
district-wide. However, compared with participants, non¬ 
participants were less likely to enjoy learning how things 
work or to report that engineers mainly work with other 
people to solve problems. Our qualitative evidence under¬ 
scores the role of coaches and industry mentors as key 
sources of information about engineering opportunities for 
participants. Therefore, although nonparticipants moderately 
raised their educational expectations, they did not share in 
the trend among participants of focusing educational 
expectations around the concrete goal of a career in 
engineering or science. This suggests that although interest 
in engineering and science careers increased for all students 
following the implementation of the Beyond Blackboards 
program, those who did not participate did not enjoy the 
benefit of increased understanding of what these careers 
were like and what is required to obtain them. 

Recall that the program takes a design-based approach to 
solving socially relevant problems. This approach could 
contribute to the outcomes we observe in several ways. As 
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evidence from the focus groups suggests, the program 
participants’ experiences in teams working on difficult 
problems that have more than one possible solution may 
have bolstered students’ confidence that they can solve 
such problems, that it is acceptable to take risks and try 
various solutions, and that working as a team can be 
rewarding. It also gave them a hands-on, experiential view 
of what an engineer does—a critical step in developing 
aligned ambitions (Schneider & Stevenson, 1999). 

The early success of the FLL teams may have 
contributed to the school-wide shift in attitudes about 
STEM. Teachers, students, administrators, parents, and 
community leaders all took note and were proud of the 
teams’ successes. Successes were featured in the local 
news, on school and district web sites, and in school-wide 
assemblies. This attention may have contributed to the 
positive shift in students’ attitudes about STEM that we 
observed even among students who did not participate. 

Although our results are consistent with the idea that 
incorporating design principles shapes students’ engineer¬ 
ing knowledge, ambitions, and plans as well as the school¬ 
wide climate surrounding engineering, our findings are 
suggestive rather than conclusive about the cause of the 
observed changes. Further evidence is needed to ascertain 
whether and how Beyond Blackboards has an effect on 
participants and their schoolmates and to determine why it 
contributes to changes in attitudes, interests, and expecta¬ 
tions. Our comparisons between students’ fall and spring 
survey responses demonstrate overall changes among both 
participants and nonparticipants in a setting composed 
predominantly of students who are traditionally under¬ 
represented in engineering—namely, students who are 
Hispanic, economically disadvantaged, and often speak 
languages other than English at home. 

Throughout the analysis, we were sensitive to issues of 
self-selection and potential sources of selectivity among our 
respondents. For example, students who participated likely 
differed from those who did not in motivation and in the 
circumstances that enable participation, and this is reflected 
in the participants’ highly positive views toward engineer¬ 
ing in the fall. In this light, our finding that students’ 
interest in engineering increased may seem obvious; 
however, one possible alternate scenario would be that as 
students gained clearer understandings of what engineering 
entails, their interest in engineering could have declined. 
Still, we wanted to be sure that selectivity into participation 
was not driving our results and that our findings would hold 
true for minority and low-income students. As a feasible 
robustness check, when we restricted our analysis to 
the subsample of Hispanic students and to economically 
disadvantaged students, the results were consistent with our 
conclusions. Our sample size precluded such testing for 
girls, English-language learners, and other minorities, but 
we encourage future research in this area. However, this 
preliminary check suggests that the positive association 


between participation and attitudes toward engineering and 
careers in engineering is shared by underrepresented 
students and their more privileged classmates alike. 

It may be helpful for future investigations to extend this 
line of inquiry further as well as identify the extent to which 
students benefit from team membership in general versus 
particular roles and experiences. For example, Beyond 
Blackboards could affect students differently depending on 
whether they assumed a leadership role in a large team or 
worked in a smaller, more egalitarian group. Nonetheless, 
we have provided important new evidence for the value of 
programs such as Beyond Blackboards', in particular, we 
observe a striking school-wide shift in STEM-related 
interest, and we did not observe a concurrent shift in more 
general attitudes about school. We collected the data for 
this analysis during the first year of the program, which 
brings with it certain limitations, such as small sample size 
and a smaller window of time to observe changes among 
Cave Creek students. These limitations were necessary, 
however, because the first year of implementation is the 
only window of opportunity to observe any shift in school¬ 
wide culture from the introduction of a new program. 

The CCSD provides an excellent venue for an interven¬ 
tion such as Beyond Blackboards to take hold broadly 
across all middle schools in the district and contribute to 
promoting diversity in our future engineering labor force. 
As a relatively small, rural, and tightly knit community of 
working-class and poor families with limited education, the 
Cave Creek community cares about providing opportunities 
for its children. Beyond Blackboards is broadly appeal¬ 
ing to this community because it encourages students to 
consider a future in the STEM fields; aligns students’ 
ambitions; and brings together students, teachers, and 
families around activities that connect to future educational 
and career opportunities. 
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Appendix 

A. Beyond Blackboards: Engineering and curricular aspects of the program. 

Foundations of engineering education approach 

Beyond Blackboards curriculum 

Principles of K-12 engineering (National Academy of Engineering & 
National Resource Council, 2009) 

• An emphasis on engineering design and the design 

process 

• Incorporation of important and developmentally 
appropriate mathematics, science, and technology 
knowledge and skills 

• Promotion of engineering habits of mind: 

(1) systems thinking, (2) creativity, (3), optimism, 

(4) collaboration, (5) ethical considerations 

Based on the 21st Century’s Grand Engineering Challenges (National 
Academy of Engineering, 2008) 

Themes: sustainability, health, reducing vulnerability, joy of living 

Example units: 

• “Engineering entrepreneurship” where students formed Grand Challenge 
companies 

• “Relay for a Reason” where students applied robotics solutions to 

natural disaster relief 

• “Biomedical Outreach and Leadership Team” (BOLT) where students 
learned about biomedical design innovations 

Mode of learning: design-based, project-focused, hands-on activities 
incorporating applied mathematics and science 

Extension to careers: STEM Stations included in each meeting emphasizing 
real-world applications of disciplinary integration of science, technology, 
engineering, and mathematics 

Source: Engineering in K-12 education: Understanding the status and 
improving the prospects (National Academy of Engineering & 

National Resource Council, 2009). 

Source: Changing the conversation: Messages for improving public 

understanding of engineering (National Academy of Engineering, 2008). 







